Dynamic memory management in workload acceleration

ABSTRACT

An apparatus for selecting a memory management method includes a memory condition module that determines memory parameters of host memory and device memory prior to a device executing a function and a memory selection module that selects a memory management method based on the memory parameters. The apparatus includes and an implementation module that implements the selected memory management method in association with execution of the function.

FIELD

The subject matter disclosed herein relates to memory management andspecifically to selecting a memory management method based on a functioncall.

BACKGROUND

When executing a program, it is common for the program to include afunction call. Typically, the function call is coded to include a callto a specific function executing on as specific processor. The functionis typically part of a library of functions of a processor, such as agraphical processing unit (“GPU”), accelerator, etc. A function oftenexecutes a particular type of task. For example, a function may be forexecution of a fast Fourier transform (“FFT”), a linear regression, orother calculation-intensive function. However, the specified functionand associated processor may not be the best choice and other availablefunctions and processors may be a better choice. Depending on thefunction, a memory management method other than a current memorymanagement method may also be a better choice.

BRIEF SUMMARY

An apparatus for selecting a memory management method is disclosed. Amethod and computer program product also perform the functions of theapparatus. The apparatus includes a memory condition module thatdetermines memory parameters of host memory and device memory prior to adevice executing a function and a memory selection module that selects amemory management method based on the memory parameters. The apparatusincludes and an implementation module that implements the selectedmemory management method in association with execution of the function.

In one embodiment, the memory implementation module changes from acurrent memory management method to the selected memory managementmethod. In another embodiment, the memory management methods includepageable, page-locked, and one of zero-copy and unified address spacememory management methods. In another embodiment, the memory parametersinclude amount of available host memory, amount of available devicememory, number of times a block of memory is accessed, and/or acomplexity of memory allocation based on commands in a queue.

In another embodiment, the memory selection module selects the pageablememory management method in response to the memory condition moduledetermining that the available host memory is below a host memorythreshold, the page-locked memory management method in response to thememory condition module determining that the available host memory isabove the host memory threshold and one or more of the available devicememory is above a device memory threshold, a same memory block in thehost memory is accessed more than a memory block access threshold andthe complexity of memory allocation is below a complexity threshold, andthe zero-copy memory management method or the unified address spacememory management method in response to the memory condition moduledetermining that the available host memory is above the host memorythreshold and one or more of the available device memory is below thedevice memory threshold, a same memory block in the host memory isaccessed less than a memory block access threshold and the complexity ofmemory allocation is above a complexity threshold.

In one embodiment, the memory condition module also includes a memoryhistory module that determines memory parameters present duringexecution of one or more executed functions, and/or a current conditionmodule that determines current memory parameters of the host memory andthe device memory. In another embodiment, the memory history moduletracks, along with the memory parameters of an executed function,function type, function parameters, and/or which device executed thefunction, wherein a plurality of devices were available for execution ofthe function.

In another embodiment, each memory management method includes aweighting factor and the memory condition module further adjusts theweighting factor of each memory management method based on thedetermined memory parameters and the memory selection module selects thememory management method based on the weighting factors. In anotherembodiment, each memory management method includes a historicalweighting factor and a current conditions weighting factor and thememory history module adjusts the historical weighting factor of eachmemory management method based on memory parameters present duringexecution of the one or more previously executed functions, and thecurrent condition module adjusts the current conditions weighting factorof each memory management method based on current memory parameters ofthe host memory and the device memory. The memory condition modulecombines the historical weighting factor and the current conditionsweighting factor for each memory management method.

In one embodiment, the function is specified in a function call issuedfrom an application executing on the host. In another embodiment, thedevice includes one of a plurality of devices available for execution ofthe function. In another embodiment, the memory condition moduledetermines the memory parameters in response to selection of a devicefor execution of the function.

A method for selecting a memory management method includes determiningmemory parameters of host memory and device memory prior to a deviceexecuting a function, and selecting a memory management method based onthe memory parameters. In another embodiment, implementing the selectedmemory management method is in association with execution of thefunction. In another embodiment, the memory management methods includepageable, page-locked, and one of zero-copy and unified address spacememory management methods. In another embodiment, the memory parametersinclude amount of available host memory, amount of available devicememory, number of times a page within the host memory is accessed,and/or a complexity of memory allocation based on commands in a queue.

In another embodiment, selecting a memory management method includesselecting the pageable memory management method in response to thememory condition module determining that the available host memory isbelow a host memory threshold, the page-locked memory management methodin response to the memory condition module determining that theavailable host memory is above the host memory threshold and theavailable device memory is above a device memory threshold, a same pagein the host memory is accessed more than a page access threshold and/orthe complexity of memory allocation is below a complexity threshold, andthe zero-copy memory management method or the unified address spacememory management method in response to the memory condition moduledetermining that the available host memory is above the host memorythreshold and the available device memory is below the device memorythreshold, a same page in the host memory is accessed less than a pageaccess threshold and/or the complexity of memory allocation is above acomplexity threshold.

In one embodiment, the method includes determining memory parameterspresent during execution of a plurality of previously executedfunctions, and determining current memory parameters of the host memoryand the device memory. In another embodiment, determining memoryparameters present during execution of a plurality of previouslyexecuted functions includes tracking, along with the memory parametersof an executed function, function type, function parameters, and/orwhich device executed the function, where a plurality of devices wereavailable for execution of the function.

In another embodiment, each memory management method includes aweighting factor and determining memory parameters further includesadjusting the weighting factor of each memory management method based onthe determined memory parameters and selecting a memory managementmethod comprises selecting the memory management method based on theweighting factors.

In another embodiment, each memory management method includes ahistorical weighting factor and a current conditions weighting factorand determining memory parameters present during execution of aplurality of previously executed functions includes adjusting thehistorical weighting factor of each memory management method based onmemory parameters present during execution of the plurality ofpreviously executed functions and determining current memory parametersof the host memory and the device memory includes adjusting the currentconditions weighting factor of each memory management method based oncurrent memory parameters of the host memory and the device memory.Determining the memory parameters includes combining the historicalweighting factor and the current conditions weighting factor for eachmemory management method.

A system for selecting a memory management method includes a host with acentral processing unit, a host memory accessible by the host, a devicewith a processor, a processor memory accessible by the device, and acommunication bus between the host, the host memory, the device and/orthe device memory. The system includes a memory condition module thatdetermines memory parameters of the host memory and the device memoryprior to the device executing a function, a memory selection module thatselects a memory management method based on the memory parameters, and amemory implementation module that implements the selected memorymanagement method in association with execution of the function. z,999

BRIEF DESCRIPTION OF THE DRAWINGS

A more particular description of the embodiments briefly described abovewill be rendered by reference to specific embodiments that areillustrated in the appended drawings. Understanding that these drawingsdepict only some embodiments and are not therefore to be considered tobe limiting of scope, the embodiments will be described and explainedwith additional specificity and detail through the use of theaccompanying drawings, in which:

FIG. 1 is a schematic block diagram illustrating one embodiment of asystem for selecting a processor for executing a function call;

FIG. 2 is a schematic block diagram illustrating one embodiment of anapparatus for intercepting a function call;

FIG. 3 is a schematic block diagram illustrating one embodiment of analternate apparatus for intercepting a function call;

FIG. 4 a schematic flow chart diagram illustrating one embodiment of amethod for intercepting and rerouting a function call for execution;

FIG. 5 is a schematic flow chart diagram illustrating one embodiment ofa method for intercepting a function call;

FIG. 6 is a schematic flow chart diagram illustrating one embodiment ofan alternate method for intercepting a function call;

FIG. 7 is a schematic block diagram illustrating one embodiment of anapparatus for determining execution of a function call based onefficiency;

FIG. 8 is a schematic block diagram illustrating one embodiment of analternate apparatus for determining execution of a function call basedon efficiency;

FIG. 9 is a schematic flow chart diagram illustrating one embodiment ofa method for determining execution of a function call based onefficiency;

FIG. 10 is a schematic flow chart diagram illustrating one embodiment ofan alternate method for determining execution of a function call basedon efficiency;

FIG. 11 is a chart depicting one embodiment of incremental energyrequirements of three processors based on workload size;

FIG. 12 is a chart depicting another embodiment of incremental energyrequirements of three processors based on workload size;

FIG. 13 is a schematic block diagram illustrating one embodiment of anapparatus for determining execution of a function call based onperformance;

FIG. 14 is a schematic block diagram illustrating one embodiment of analternate apparatus for determining execution of a function call basedon performance;

FIG. 15 is a schematic flow chart diagram illustrating one embodiment ofa method for determining execution of a function call based onperformance;

FIG. 16 is a schematic flow chart diagram illustrating one embodiment ofan alternate method for determining execution of a function call basedon performance;

FIG. 17 is a chart depicting one embodiment of computing speed versusfast Fourier transform size of two processors;

FIG. 18 is a schematic block diagram illustrating one embodiment of anapparatus for determining a memory management method;

FIG. 19 is a schematic block diagram illustrating one embodiment of analternate apparatus for determining a memory management method;

FIG. 20 is a schematic flow chart diagram illustrating one embodiment ofa method for determining a memory management method;

FIG. 21A is a first part of a schematic flow chart diagram illustratingone embodiment of an alternate method for determining a memorymanagement method; and

FIG. 21B is a second part of a schematic flow chart diagram illustratingone embodiment of the alternate method for determining a memorymanagement method.

DETAILED DESCRIPTION

As will be appreciated by one skilled in the art, aspects of theembodiments may be embodied as a system, method or program product.Accordingly, embodiments may take the form of an entirely hardwareembodiment, an entirely software embodiment (including firmware,resident software, micro-code, etc.) or an embodiment combining softwareand hardware aspects that may all generally be referred to herein as a“circuit,” “module” or “system.” Furthermore, embodiments may take theform of a program product embodied in one or more computer readablestorage devices storing machine readable code, computer readable code,and/or program code, referred hereafter as code. The storage devices maybe tangible, non-transitory, and/or non-transmission. The storagedevices may not embody signals. In a certain embodiment, the storagedevices only employ signals for accessing code.

Many of the functional units described in this specification have beenlabeled as modules, in order to more particularly emphasize theirimplementation independence. For example, a module may be implemented asa hardware circuit comprising custom VLSI circuits or gate arrays,off-the-shelf semiconductors such as logic chips, transistors, or otherdiscrete components. A module may also be implemented in programmablehardware devices such as field programmable gate arrays, programmablearray logic, programmable logic devices or the like.

Modules may also be implemented in code and/or software for execution byvarious types of processors. An identified module of code may, forinstance, comprise one or more physical or logical blocks of executablecode which may, for instance, be organized as an object, procedure, orfunction. Nevertheless, the executables of an identified module need notbe physically located together, but may comprise disparate instructionsstored in different locations which, when joined logically together,comprise the module and achieve the stated purpose for the module.

Indeed, a module of code may be a single instruction, or manyinstructions, and may even be distributed over several different codesegments, among different programs, and across several memory devices.Similarly, operational data may be identified and illustrated hereinwithin modules, and may be embodied in any suitable form and organizedwithin any suitable type of data structure. The operational data may becollected as a single data set, or may be distributed over differentlocations including over different computer readable storage devices.Where a module or portions of a module are implemented in software, thesoftware portions are stored on one or more computer readable storagedevices.

Any combination of one or more computer readable medium may be utilized.The computer readable medium may be a computer readable storage medium.The computer readable storage medium may be a storage device storing thecode. The storage device may be, for example, but not limited to, anelectronic, magnetic, optical, electromagnetic, infrared, holographic,micromechanical, or semiconductor system, apparatus, or device, or anysuitable combination of the foregoing.

More specific examples (a non-exhaustive list) of the storage devicewould include the following: an electrical connection having one or morewires, a portable computer diskette, a hard disk, a random access memory(“RAM”), a read-only memory (“ROM”), an erasable programmable read-onlymemory (“EPROM” or “Flash memory”), a portable compact disc read-onlymemory (“CD-ROM”), an optical storage device, a magnetic storage device,or any suitable combination of the foregoing. In the context of thisdocument, a computer readable storage medium may be any tangible mediumthat can contain, or store a program for use by or in connection with aninstruction execution system, apparatus, or device.

Code for carrying out operations for embodiments may be written in anycombination of one or more programming languages including an objectoriented programming language such as Python, Ruby, Java, Smalltalk,C++, or the like, and conventional procedural programming languages,such as the “C” programming language, or the like, and/or machinelanguages such as assembly languages. The code may execute entirely onthe user's computer, partly on the user's computer, as a stand-alonesoftware package, partly on the user's computer and partly on a remotecomputer or entirely on the remote computer or server. In the latterscenario, the remote computer may be connected to the user's computerthrough any type of network, including a local area network (LAN) or awide area network (WAN), or the connection may be made to an externalcomputer (for example, through the Internet using an Internet ServiceProvider).

Reference throughout this specification to “one embodiment,” “anembodiment,” or similar language means that a particular feature,structure, or characteristic described in connection with the embodimentis included in at least one embodiment. Thus, appearances of the phrases“in one embodiment,” “in an embodiment,” and similar language throughoutthis specification may, but do not necessarily, all refer to the sameembodiment, but mean “one or more but not all embodiments” unlessexpressly specified otherwise. The terms “including,” “comprising,”“having,” and variations thereof mean “including but not limited to,”unless expressly specified otherwise. An enumerated listing of itemsdoes not imply that any or all of the items are mutually exclusive,unless expressly specified otherwise. The terms “a,” “an,” and “the”also refer to “one or more” unless expressly specified otherwise.

Furthermore, the described features, structures, or characteristics ofthe embodiments may be combined in any suitable manner. In the followingdescription, numerous specific details are provided, such as examples ofprogramming, software modules, user selections, network transactions,database queries, database structures, hardware modules, hardwarecircuits, hardware chips, etc., to provide a thorough understanding ofembodiments. One skilled in the relevant art will recognize, however,that embodiments may be practiced without one or more of the specificdetails, or with other methods, components, materials, and so forth. Inother instances, well-known structures, materials, or operations are notshown or described in detail to avoid obscuring aspects of anembodiment.

Aspects of the embodiments are described below with reference toschematic flowchart diagrams and/or schematic block diagrams of methods,apparatuses, systems, and program products according to embodiments. Itwill be understood that each block of the schematic flowchart diagramsand/or schematic block diagrams, and combinations of blocks in theschematic flowchart diagrams and/or schematic block diagrams, can beimplemented by code. These code may be provided to a processor of ageneral purpose computer, special purpose computer, or otherprogrammable data processing apparatus to produce a machine, such thatthe instructions, which execute via the processor of the computer orother programmable data processing apparatus, create means forimplementing the functions/acts specified in the schematic flowchartdiagrams and/or schematic block diagrams block or blocks.

The code may also be stored in a storage device that can direct acomputer, other programmable data processing apparatus, or other devicesto function in a particular manner, such that the instructions stored inthe storage device produce an article of manufacture includinginstructions which implement the function/act specified in the schematicflowchart diagrams and/or schematic block diagrams block or blocks.

The code may also be loaded onto a computer, other programmable dataprocessing apparatus, or other devices to cause a series of operationalsteps to be performed on the computer, other programmable apparatus orother devices to produce a computer implemented process such that thecode which execute on the computer or other programmable apparatusprovide processes for implementing the functions/acts specified in theflowchart and/or block diagram block or blocks.

The schematic flowchart diagrams and/or schematic block diagrams in theFigures illustrate the architecture, functionality, and operation ofpossible implementations of apparatuses, systems, methods and programproducts according to various embodiments. In this regard, each block inthe schematic flowchart diagrams and/or schematic block diagrams mayrepresent a module, segment, or portion of code, which comprises one ormore executable instructions of the code for implementing the specifiedlogical function(s).

It should also be noted that, in some alternative implementations, thefunctions noted in the block may occur out of the order noted in theFigures. For example, two blocks shown in succession may, in fact, beexecuted substantially concurrently, or the blocks may sometimes beexecuted in the reverse order, depending upon the functionalityinvolved. Other steps and methods may be conceived that are equivalentin function, logic, or effect to one or more blocks, or portionsthereof, of the illustrated Figures.

Although various arrow types and line types may be employed in theflowchart and/or block diagrams, they are understood not to limit thescope of the corresponding embodiments. Indeed, some arrows or otherconnectors may be used to indicate only the logical flow of the depictedembodiment. For instance, an arrow may indicate a waiting or monitoringperiod of unspecified duration between enumerated steps of the depictedembodiment. It will also be noted that each block of the block diagramsand/or flowchart diagrams, and combinations of blocks in the blockdiagrams and/or flowchart diagrams, can be implemented by specialpurpose hardware-based systems that perform the specified functions oracts, or combinations of special purpose hardware and code.

The description of elements in each figure may refer to elements ofproceeding figures. Like numbers refer to like elements in all figures,including alternate embodiments of like elements.

FIG. 1 depicts a schematic block diagram illustrating one embodiment ofa system 100 for selecting a processor for executing a function call.The system includes an intercept apparatus 102, an efficiency apparatus104, a performance apparatus 106, and a memory apparatus 108 in acentral processing unit (“CPU”) memory 110, one or more CPUs 112, anorthbridge 114, a graphics processing unit (“GPU”) 116, a GPU memory118, a field-programmable gate array (“FPGA”) 120, an FPGA memory 122,an accelerator 124, an accelerator memory 126, a digital signalprocessor (“DSP”), a DSP memory 130, a southbridge 132, one or more datastorage devices 134, one or more peripherals 136, a memory bus 138, afrontside bus (“FSB”) 140, a high-speed bus 142, and an internal bus144, which are described below.

In one embodiment, the system 100 includes an intercept apparatus 102that dynamically intercepts a function call from an executingapplication, selects a function from several available functions, andreroutes the function call to the selected function. The interceptapparatus 102 is described in more detail with regard to the apparatuses200, 300 of FIGS. 2 and 3.

In one embodiment, the system 100 includes an efficiency apparatus 104that compares energy consumption characteristics of several processorsavailable for execution of a function where the energy consumptioncharacteristic varies based on function size, selects, based on size ofthe function, a processor with a lowest energy consumption, and executesthe function on the selected processor. The efficiency apparatus 104 isdescribed in more detail with regard to the apparatuses 700, 800 ofFIGS. 7 and 8.

In another embodiment, the system 100 includes a performance apparatus106 that compares performance characteristics of several processorsavailable for execution of a function where the performancecharacteristics vary based on function size, that selects, based on thesize of the function, a processor for execution of the function, andthat executes the function on the selected processor. The performanceapparatus 106 is described in more detail with regard to the apparatuses1300, 1400 of FIGS. 13 and 14.

In another embodiment, the system 100 includes a memory apparatus 108that determines memory parameters of host memory and device memory priorto executing a function, selects a memory management method based on thememory parameters, and implements the selected memory management methodin association with execution of the function. The memory apparatus 108is described in more detail with regard to the apparatuses 1800, 1900 ofFIGS. 18 and 19.

The system 100 includes CPU memory 110 that includes, in one embodiment,the intercept apparatus 102, the efficiency apparatus 104, theperformance apparatus 106, and the memory apparatus 108. While theapparatuses 102, 104, 106, 108 are depicted in the CPU memory 110, oneof skill in the art will recognize that all or a portion of theapparatuses 102, 104, 106, 108 may be located elsewhere in the system100 and may include hardware circuits. The apparatuses 102, 104, 106,108 may be implemented in one or more hardware circuits, programmablelogic devices, and/or executable code stored on computer readablestorage media.

The CPU memory 110 may be a single memory as depicted or may includemultiple memory types, levels, etc. For example, each CPU 112 may have aseparate CPU memory 110 and the system 100 may also share another CPUmemory 110. In one embodiment, the CPU memory 110 connects to the one ormore CPUs 112 over a memory bus 138 through the northbridge 114. One ofskill in the art will recognize other forms of CPU memory 110.

The system 100, in one embodiment, includes a northbridge 114 incommunication with the CPUs 112, the CPU memory 110, as well asavailable processors, such as a GPU 116, an FPGA 120, an accelerator 124and a DSP 128. The northbridge 114 may also be in communication with asouthbridge 132 as well as other elements of the system 100 that are notshown. The northbridge 114, in one embodiment, the northbridge 114includes a chipset and connects directly to the one or more CPUs 112 andCPU memory 110 over a frontside bus (“FSB”) 140 and typicallycommunicates at a speed higher than other components connected to thesouthbridge 132. The northbridge 114, in one embodiment, facilitatesconnection to alternate processors 116, 120, 124, 128 available toexecute a function, such as the GPU 116, accelerator 124, etc.Typically, the northbridge 114 connects to the alternate processors 116,120, 124, 128 over one or more high-speed busses 142. The high-speedbusses 142 may include a Peripheral Component Interconnect Express (“PCIExpress®” or “PCIe”) bus, an Accelerated Graphics Port (“AGP”), etc.

In one embodiment, one or more alternate processors 116, 120, 124, 128are collocated with the one or more CPUs 112. In another embodiment, oneor more alternate processors 116, 120, 124, 128 are located remote fromthe CPUs 112. While a single GPU 116, FPGA 120, accelerator 124 and DSP128 are depicted in FIG. 1, one of skill in the art will recognize otherconfigurations that may include more or less of each processor typewhere one or more of the alternate processors 116, 120, 124, 128 arecollocated with the CPUs 112 and/or one or more of the alternateprocessors 116, 120, 124, 128 are remote from the CPUs 112.

Where an alternate processor (e.g. the FPGA 120) is remote from the CPUs112, the alternate processor may be connected over a network or ahigh-speed bus 142. The network may include one or more of a local areanetwork (“LAN”), a wide area network (“WAN”), the Internet, a wirelessnetwork, a fiber optic network, etc. The network may be connected invarious ways, such as through the southbridge 132 over a networkinterface card.

The wireless network may be a mobile telephone network. The wirelessnetwork may also employ a Wi-Fi network based on any one of theInstitute of Electrical and Electronics Engineers (“IEEE”) 802.11standards. Alternatively, the wireless network may be a BLUETOOTH®connection. In addition, the wireless network may employ a RadioFrequency Identification (“RFID”) communication including RFID standardsestablished by the International Organization for Standardization(“ISO”), the International Electrotechnical Commission (“IEC”), theAmerican Society for Testing and Materials® (ASTM®), the DASH7™Alliance, and EPCGlobal™

Alternatively, the wireless network may employ a ZigBee® connectionbased on the IEEE 802 standard. In one embodiment, the wireless networkemploys a Z-Wave® connection as designed by Sigma Designs®.Alternatively, the wireless network may employ an ANT® and/or ANT+®connection as defined by Dynastream® Innovations Inc. of Cochrane,Canada.

The wireless network may be an infrared connection including connectionsconforming at least to the Infrared Physical Layer Specification(“IrPHY”) as defined by the Infrared Data Association® (IrDA®).Alternatively, the wireless network may be a cellular telephone networkcommunication. All standards and/or connection types include the latestversion and revision of the standard and/or connection type as of thefiling date of this application.

As depicted for the alternate processors 116, 120, 124, 128, the GPU 116is in communication with GPU memory 118, the FPGA 120 is incommunication with FPGA memory 122, the accelerator 124 is communicationwith accelerator memory 126 and the DSP 128 is in communication with theDSP memory 130. In alternate embodiments, one or more of the alternateprocessors 116, 120, 124, 128 do not include memory and rely on the CPUmemory 110 or other memory. For example, a GPU 116 may not includememory and may rely on the CPU memory 110.

The GPU 116, in one embodiment, is capable of executing graphicaldisplay commands to drive an electronic display (not shown), but is alsocapable of executing other functions. For example, the GPU 116 may be ageneral purpose GPU (“GPGPU”) capable of executing functions in additionto execution of typical graphics commands. In one embodiment, the GPU116 supports a language that facilitates execution of various functions.For example, the GPU 116 may support application program interface(“API”) extensions to the C programming language, such as CUDA® fromNvidia® Corporation and OpenCL® from Apple® Corporation.

The FPGA 120 may be configured to perform complex combinationalfunctions and may be specifically designed for specific function types.The FPGA 120, for example, may be designed for computationally intensivefunctions and may, by their nature, be faster than the CPUs 112 forcertain functions, or may be more power efficient for certain functionsthan the CPUs 112.

The DSP 128 may be included in the system 100 and may include anarchitecture optimized for the operational needs of digital signalprocessing functions. The DSP 128 may measure, filter and/or compressanalog signals. The DSP 128 may also be used for execution of certainfunctions and may be more efficient or faster for execution of aparticular function than the CPUs 112 or other of the alternateprocessors 116, 120, 124.

The accelerator 124 may include hardware acceleration to perform somefunctions faster than the CPUs 112 or other of the alternate processors116, 120, 128. In some embodiments, an accelerator 124 may be a generalcategory that includes a GPU 116, a DSP 128, etc. or may be designed forsome other hardware acceleration tasks. An accelerator 128 may bedesigned for computationally intensive software code. One of skill inthe art will recognize other features and advantages of various types ofalternate processors 116, 120, 124, 128 and will recognize other typesof alternate processors 116, 120, 124, 128 that may be used forexecution of a function.

Each of the alternate processors 116, 120, 124, 128 is capable ofexecuting a function. For example, an executing program may issue afunction call and the intercept apparatus 102 may intercept the functioncall and may determine which of the available alternate processors 116,120, 124, 128 is capable of executing the function and may determine abest choice for executing the function. Determining which of thealternate processors 116, 120, 124, 128 to select may be made based onpower efficiency, performance, workload, etc. A particular function callmay select a function and a processor for executing the function. In oneembodiment, the intercept apparatus 102 intercepts the function call anddetermines dynamically which of the alternate processors 116, 120, 124,128 is a best choice for execution of the function, which may be adifferent processor than coded into the function call.

The system 100, in one embodiment, includes a southbridge 132 connectedto the northbridge 114 over an internal bus 144. The southbridge 132 mayinclude a chipset that includes functionality to connect to peripherals136, data storage devices 134, and the like. The southbridge 132 mayconnect using various connection types, such as a universal serial bus(“USB”), a parallel ATA bus, a serial ATA bus, an IEEE 1394 bus, and thelike. One of skill in the art will recognize equipment that may connectover a southbridge 132 and connection types.

FIG. 2 is a schematic block diagram illustrating one embodiment of anapparatus 200 for intercepting a function call. The apparatus 200includes one embodiment of the intercept module 202 that includes anintercept module 202, a selection module 204, and an execution module206, which are described below. While the modules 202, 204, 206 aredepicted in the intercept apparatus 102, in some embodiments, some orall of the modules 202, 204, 206 may be located elsewhere and may beimplemented in various ways, such as hardware circuits, executable code,etc. and may be implemented using a combination of executable codecombined with hardware circuits.

The apparatus 200, in one embodiment, includes an intercept module 202that intercepts a function call from an executing application. Thefunction call, in one embodiment, specifies a target function where thetarget function is associated with a target processor. For example, thetarget processor may be a CPU 112 or an alternate processor 116, 120,124, 128.

Typically, a function call is a static call where the function isspecified along with various other parameters for execution. As part ofthe static function call, the target processor is specified. However,the target processor may not be the best choice for execution of thefunction in terms of some desired criteria, such as low power operation,performance, etc. The intercept module 202 dynamically intercepts thefunction call during execution of the program making the function call.In one embodiment, the function call is a static call with anenvironment variable that references the target function and theintercept module 202 halts reading of the environment variable todetermine the target function and substitutes a function call thatincludes the selection module 204 and/or the execution module 206 tosubstitute the selected function for execution of the function call. Forexample, the substitute function call may call the selection module 204and/or the execution module 206, which may then select an alternatefunction to execute in place of the function specified in the originalfunction call.

As used herein, intercepting the function call refers to haltingexecution of the function after the function call but before executionof the function rather than allowing execution of the function toproceed as intended without determining if another processor would be abetter choice for satisfying the function call. In one embodiment,intercepting the function call includes halting execution of thefunction call while the selection module 204 and the execution module206 act to select and execute a function, as described below.

The apparatus 200, in one embodiment, includes a selection module 204that selects a function from a plurality of available functions toexecute the function call. In one embodiment, each available function isassociated with a separate processor (e.g. 112, 116, 120, 124, 128). Inother embodiments, each processor 112, 116, 120, 124, 128 is capable ofexecuting particular functions where the functions may overlapprocessors. For example, a CPU 112 may be able to execute a particulartype of fast Fourier transform (“FFT”) function while the GPU 116 andaccelerator 124 may also be capable of execution of the same function orsame function type. The selection module 204 may select between FFTfunctions for the CPU 112, the GPU 116 and the accelerator 124 to selecta function and associated processor that is most appropriate for meetingcertain execution goals.

Other types of functions may include a mathematical function, linearregression, a lower level matrix operation, performing a neural networkevaluation, performing a convolutional operation, and the like. One ofskill in the art will recognize other functions that may be subject of afunction call.

In one embodiment, the selection module 204 determines characteristicsof the function call and uses the characteristics of the function callto select a function for execution of the function call. For example,the characteristics of the function call may include size of thefunction, context, function type, runtime variables, and the like. Theselection module 204 may then use characteristics of the function callto determine which function and associated processor is best suited forsatisfying the function call.

In another embodiment, the selection module 204 selects the selectedfunction based on selection of a function of the plurality of availablefunctions projected to have a highest performance based oncharacteristics of the function call. For example, for a particularfunction that processes complex linear algebra functions as well assizes of matrices in the function call, a projected number of matrixoperations, etc., the selection module 204 may select a function andassociated processor that minimizes power consumption for thatparticular function call.

In another embodiment, the selection module 204 selects the selectedfunction based on selection of a function of the plurality of availablefunctions projected to have a highest performance based oncharacteristics of the function call. For example, execution of aparticular function based on characteristics of the function call may befaster than execution of a similar function on a different availableprocessor and the selection module 204 may select the function thatexecutes fastest.

In another embodiment, the selection module 204 selects the selectedfunction based on current loading and/or a projected loading of eachprocessor associated with each of the plurality of available functions.For example, the selection module 204 may assess current operatingstatus of each available processor 112, 116, 120, 124, 128 and maydetermine that one or more of the available processors 112, 116, 120,124, 128 is busy executing another function or program. Status of eachof the available processors 112, 116, 120, 124, 128 may be a factoralong with other criteria, such as maximum efficiency, maximumperformance, etc. when selecting a function or may be a factor thateliminates a function. For example, an available processor (e.g. the GPU116) may be executing a function when the intercept module 202intercepts the function call. The selection module 204 may estimate acompletion time of execution of a function that will satisfy thefunction call by adding estimated execution time of the function plusexecution of the currently executing function.

In one embodiment, the selected function selected by the selectionmodule 204 differs from the target function. For example, the targetfunction may be scheduled to execute on the CPU 112 while the selectedfunction executes on the DSP 128. In another embodiment, the selectionmodule 204 determines that the target function is most appropriate andthe target function becomes the selected function so that the functioncall executes as originally stated.

The apparatus 200, in one embodiment, includes an execution module 206that reroutes the function call to execute the selected function. Theexecution module 206 typically reroutes the function call so that theselected function executes on an associated processor. For example, theselected function may execute on the GPU 116 instead of one of the CPUs112 so that the execution module 206 reroutes the function call toexecute the selected function on the GPU 116. For instance, the functioncall may specify a function on one of the CPUs 112 that includes linearregression and the selected function also includes linear regression sothat execution of the function on the GPU 116 using parameters of thefunction call generates similar or identical results as the functionthat would have executed on the selected CPU 112.

The execution module 206 rerouting the function call may includesubstituting the selected function and associated parameters in place ofthe target function and associated parameters in the function call. Inanother embodiment, the execution module 206 controls execution of theselected function, for example by issuing one or more commands to theprocessor associated with the selected function. In another embodiment,the execution module 206 returns results of the executed function to theexecuting program. One of skill in the art will recognize otherfunctions of the execution module 206 for executing the selectedfunction in place of the target function.

FIG. 3 is a schematic block diagram illustrating one embodiment of analternate apparatus 300 for intercepting a function call. The apparatus300 includes another embodiment, of the intercept apparatus 102 with anintercept module 202, a selection module 204, and an execution module206 that are substantially similar to those described above in relationto the apparatus 200 of FIG. 2. In some embodiments, the apparatus 300also includes a reformat module 302 and/or a return module 304, whichare described below.

In one embodiment, the function call is formatted for execution of thetarget function on the processor associated with the target function.For example, the function call may be formatted for execution on one ofthe CPUs 112. In the embodiment, the apparatus 300 includes a reformatmodule 302 that reformats the function call for execution of theselected function on the processor associated with the selectedfunction. For example, the selected function may execute on the GPU 116and the reformat module 302 reformats the function call to call theselected function for execution on the GPU 116. The reformat module 302,for example, may adjust parameters, variables, executable code, and thelike so that the selected function executes and provides the sameresults or similar results to results that would have been produced byexecuting the target function.

In another embodiment, the apparatus 300 includes a return module 304that formats results from the selected function in a format compatiblewith a format of results from the target function and returns theresults to the application. For example, if the selected functionexecutes on the GPU 116, results may be in a format different than fromthe target function and the return module 304 may reformat the resultsfrom the selected function so that results returned in response to thefunction call are in an expected format.

The reformat module 302 and the return module 304 beneficially allow auser to format a function with a standard format, for example, in aformat for one of the CPUs 112 and then the apparatus 300 is able toselect a different function and associated processor without the userbeing required to reformat the function call.

FIG. 4 is a schematic flow chart diagram illustrating one embodiment ofa method 400 for intercepting and rerouting a function call forexecution. The flow chart 400 graphically illustrates a function call402 that is written for a CPU (e.g. 112) that can be controlled on theCPU and can use the CPU interface. The apparatuses 200, 300 of FIGS. 2and 3 may dynamically intercept the function call 402 at runtime and maythen select an available function from the intermediate library 404.

Any number of available functions may substitute for the targetfunction. As depicted, some libraries may include a Math Kernel Library(“MKL”) that may include core math functions, sparse solvers, FFTs,vector math, etc. In one embodiment, functions from the MKL library mayrun on one or more of the CPUs 112. In another embodiment, an MKLlibrary may be included for execution on a different processor, such asan accelerator 124, which may include an Intel® Xeon Phi™ coprocessor(depicted in FIG. 4 as “Phi”). In another embodiment, the selectionmodule 204 may select a function from a graphical processor library,such as the NVIDA® CUDA® Basic Linear Algebra Subroutines (“cuBLAS”)library or similar library, for execution on the GPU 116.

In another embodiment, the selection module 204 selects a function froma library for the FPGA 120, such as the FPGA Basic Linear AlgebraSubroutines (“fBLAS”) library for execution on the FPGA 120. Thelibraries and associated processors depicted in FIG. 4 are merely forillustration and one of skill in the art will recognize that otheralternate processors and processor types may be available for executionof a function call and that other function libraries may be available.

FIG. 5 is a schematic flow chart diagram illustrating one embodiment ofa method 500 for intercepting a function call. The method 500 begins andintercepts 502 a function call from an executing application. Thefunction call specifies a target function where the target function isassociated with a target processor. In one embodiment, the interceptmodule 202 intercepts 502 the function call. The method 500 selects 504a function from among several available functions to execute thefunction call. In one example, the selection module 204 selects 504 afunction. In one embodiment, each available function is associated witha separate processor. The method 500 reroutes 506 the function call toexecute the selected function, and the method 500 ends. In oneembodiment, the execution module 206 reroutes 506 the function call toexecute the selected function.

FIG. 6 is a schematic flow chart diagram illustrating one embodiment ofan alternate method 600 for intercepting a function call. The method 600begins and gathers 602 data from previously executed functions. Forexample, the method 600 may gather 602 data from each availableprocessor regarding executed functions. The data may include executiontime, context information, function size, energy consumption, and thelike.

The method 600 intercepts 604 a function call from an executingapplication where the function call specifies a target function and thetarget function is associated with a target processor. The method 600selects 606 a function from a plurality of available functions toexecute the function call, and in one embodiment, each availablefunction is associated with a separate processor. The method 600determines 608 characteristics of the function call. For example, themethod 600 may determine 608 characteristics, such as target functionsize, target function type, parameters of the target function, and thelike. The method 600 selects 608 a function and associated processorbased on various conditions, factors, criteria, etc.

For example, the method 600 may use data from previously executedfunctions to predict execution time, energy efficiency, current load ofavailable processors, etc. for available functions and may then select608 a function that meets desired criteria, such as lowest energyconsumption, best performance, etc. In one embodiment, the selectionmodule 204 determines 606 characteristics of the function call andselects 608 a function based on the characteristics, data frompreviously executed functions, current processor conditions, etc.

The method 600 reformats 610 the function call for execution of theselected function on the processor associated with the selectedfunction. For example, the reformat module 302 may reformat 610 thefunction call. The method 600 reroutes 612 the function call to executethe selected function on the processor associated with the selectedfunction and formats 614 results from execution of the selected functionin a format compatible with a format of results from the target functionand returns 616 the results to the application, and the method 600 ends.In one embodiment, the return module 304 formats 614 results fromexecution of the selected function and returns 616 results to theapplication.

FIG. 7 is a schematic block diagram illustrating one embodiment of anapparatus 700 for determining execution of a function call based onefficiency. The apparatus 700 includes one embodiment of the efficiencyapparatus 104 that includes a comparison module 702, a selection module704, and an execution module 706, which are described below. While theefficiency apparatus 104 is depicted in the CPU memory 110 in FIG. 1 andthe modules 702, 704, 706 are depicted in the apparatus 700 in FIG. 7,one of skill in the art will recognize that all or a portion of themodules 702, 704, 706 may be located elsewhere. In addition, the modules702, 704, 706 may be implemented using hardware circuits, a programmablelogic device, executable code, or any combination thereof. For example,the comparison module 702 may include hardware circuits to measureenergy consumption or may be implemented in executable code or in aprogrammable logic device that receives power consumption data. Thelocation and makeup of the modules of the apparatus 700 of FIG. 7 mayalso apply to the modules 802, 804, 806, 808, 810, 812, 202 of theapparatus 800 of FIG. 8.

In one embodiment, the apparatus 700 includes a comparison module 702that compares energy consumption characteristics of a plurality ofprocessors available for execution of a function. Each energyconsumption characteristic varies as a function of function size. Theenergy consumption characteristics may be supplied by a vendor, providedby a user, derived from previously executed functions, etc. In oneembodiment, the energy consumption characteristic of a processor of theplurality of processors includes a startup cost and energy usage as afunction of function size.

For example, for available processor energy consumption characteristicsfor an available processor may include an initial startup cost thatincludes energy consumed as a result of preparing the processor toexecute a selected function. The processor energy consumptioncharacteristics may also include energy consumption characteristics forexecution of the selected function that vary based on function size. Ifthe processor energy consumption characteristics are expressedgraphically, a curve representing the processor energy consumptioncharacteristics may include an initial offset followed by a curve thatincreases based on function size. FIGS. 11 and 12 are representativegraphical representations of the processor energy consumptioncharacteristics of three available processors and are discussed furtherbelow.

Typically, the energy consumption of a processor increase as functionsize increases. Function size may be characterized in various ways anddepends on the function type. For example, an FFT function size may becharacterized by number of points, a linear algebra function may becharacterized by matrix size, number of matrix operations, an objectdetection function determining a number of objects in a photograph maybe characterized by the number of searched objects, number of pixels ina photograph being analyzed, and the like. One of skill in the art willrecognize function parameters that may be used to determine functionsize.

The apparatus 700, in one embodiment, includes a selection module 704that selects, based on the size of the function, a processor from theplurality of processors with a lowest energy consumption for executionof the function. For example, if a function call for the function to beexecuted is an FFT function with a particular number of points and theavailable processors that have an available FFT function are a CPU 112,a GPU 116, and an FPGA 120, the selection module 704 may use energyconsumption characteristics of the CPU 112, the GPU 116, and the FPGA120 to determine which processor is most efficient for the size of theFFT function called out in the associated function call and may theselect a function associated with this relatively most efficientprocessor.

The apparatus 700, in one embodiment, includes an execution module 706that executes the function on the selected processor. In the exampleabove, if the selection module 704 selects the GPU 116 as the mostefficient processor for the FFT function based on the function sizecalled out in the function call, the execution module 706 may execute anFFT function on the GPU 116 based on parameters from the function call.

FIG. 8 is a schematic block diagram illustrating one embodiment of analternate apparatus 800 for determining execution of a function callbased on efficiency. The apparatus 800 includes another embodiment ofthe efficiency apparatus 104 with a comparison module 702, a selectionmodule 704 and an execution module 706, which are substantially similarto those described above in relation the apparatus 700 of FIG. 7. Invarious embodiments, the efficiency apparatus 104 may also include oneor more of an efficiency module 802, an energy calculation module 804, acorrelation module 806, an update module 808, a function size module810, a startup module 812, and an intercept module 202, which aredescribed below. In addition, the selection module 704 and the executionmodule 706 may include functionality described above in relation to theapparatuses 200, 300 of FIGS. 2 and 3.

The apparatus 800, in one embodiment, includes an efficiency module 802that determines the energy consumption characteristic of each of theplurality of processors available for execution of the function. Forexample, the efficiency module 802 may access previously stored energyconsumption characteristics or may calculate an energy consumptioncharacteristic based on stored energy consumption data points.

In one embodiment, the efficiency module 802 determines the energyconsumption characteristic of each of the plurality of processors basedon information supplied with the function. For example, a user may storeenergy consumption characteristics in a library along with the functionor in a location linked to the function. In another embodiment, theefficiency module 802 determines the energy consumption characteristicof each of the plurality of processors based on information from avendor. For example, a vendor associated with a function or library offunctions that executes on a processor 112, 116, 120, 124, 128 or avendor associated with a processor 112, 116, 120, 124, 128 may provideenergy consumption characteristics associated with a function of aparticular processor. The efficiency module 802 may access the vendorsupplied energy consumption characteristics.

In another embodiment, the efficiency module 802 determines the energyconsumption characteristic of each of the plurality of processors basedon function execution data from previously executed functions. Theefficiency module 802 may access the execution data and then use theexecution data directly or may calculate energy consumptioncharacteristics for a processor based on the execution data.

In another embodiment, the apparatus 800 includes an energy calculationmodule 804 that calculates energy consumption of an executed functionexecuted on a processor of the plurality of processors and a correlationmodule 806 that correlates a size of each executed function to energyconsumed during execution of the function to determine the energyconsumption characteristic of the processor. For example, the energycalculation module 804 may determine current, voltage, power, executiontime, number of executed operations or other metric useful indetermining energy consumption and that may be associated with aprocessor executing a function and then the calculation module 804 mayuse this information to calculate energy consumption of the executedfunction.

The correlation module 806 may then determine function size for theexecuted function and may then correlate this function size with thecalculated energy consumption and may also correlate the executedfunction and associated processor with the function size and calculatedenergy consumption. The comparison module 702 may then use data providedby the efficiency module 802, the energy calculation module 804 and/orthe correlation module 806 when comparing energy consumptioncharacteristics of available processors (e.g. 112, 116, 120, 124, 128).

In one embodiment, energy consumption characteristics of availableprocessors are initially provided by a user, vendor, etc. but may beupdated with data from executed functions. In one embodiment, theapparatus 800 may include an update module 808 that updates an initialenergy consumption characteristic of a processor based on the functionexecution data from the previously executed functions.

In one embodiment, the update module 808 updates energy consumptioncharacteristics incrementally. For example, after a function is firstexecuted, only a single point is available for the update module 808 touse to update initial energy consumption characteristics and the initialenergy consumption characteristics may be used for most points along anenergy consumption curve. Over time, additional data from executedfunctions may replace initial energy consumption characteristics from avendor or user partially or fully. The update module 808 may use curvefitting, extrapolation, or other techniques to estimate energyconsumption between data points from executed functions and/or initialenergy consumption characteristics.

In another embodiment, the update module 808 replaces initial energyconsumption characteristics after execution of multiple functions whenexecution data is sufficient to establish new energy consumptioncharacteristics so that the initial energy consumption characteristicsare based on data from one or more executed functions instead of vendoror user supplied data. For example, the update module 808 may replace anentire initial energy consumption characteristic curve from a vendor oruser with a new curve based on data from executed functions. In anotherembodiment, the update module 808 replaces a section of data with datafrom executed functions. For example, the update module 808 may replacea section of initial energy consumption characteristics from a vendor oruser within a range of function sizes. One of skill in the art willrecognize other ways for the update module 808 to update initial energyconsumption characteristics with data from function execution data ofpreviously executed functions.

In one embodiment, the apparatus 800 includes a function size module 810that determines a size of the function. For example, the function sizemodule 810 may evaluate parameters, data, etc. called out or referencedin a function call to determine the size of the function. In someembodiments, the function size module 810 determines the size of thefunction based on a size of input data of the function, a number ofoperations to be performed in executing the function, a number of timesone or more functions are called during execution of the function, anumber of items searched for during execution of the function, and thelike. The comparison module 702 may then use the function size todetermine energy consumption characteristics associated with theparticular function size for the available processors.

In some embodiments, the energy consumption characteristic of aprocessor of the plurality of processors includes a startup cost andenergy usage as a function of function size. For a particular function,the startup cost includes an amount of energy to bring the processor toa state ready to execute the function. Beyond the startup cost, energyconsumption for the function typically varies with function size. Inlooking at a curve expressing an energy consumption of a function, aninitial offset may represent the startup cost and the slope and natureof the curve beyond the initial offset may represent change in energyconsumption of the processor varying with function size.

In one embodiment, the apparatus 800 includes a startup module 812 thatdetermines a current state of the processor that will execute thefunction and that adjusts the startup cost for the processor based thecurrent condition of the processor. For example, at the time of thefunction call, an available processor may be in various states. Forexample, the processor may be in a dormant state, which may require arelatively large amount of energy to reach a state where the processoris ready to execute the function or the processor may be just completingexecution of another function and may be in a ready state or a statethat requires a relatively small amount of energy to reach the readystate. The startup module 812 may then determine a current startup cost.For example, a current startup cost may shift an energy consumptioncharacteristic curve up or down. One of skill in the art will recognizeother ways for the startup module 812 to update startup cost of aprocessor available to execute a function.

In one embodiment, the apparatus 800 includes an intercept module 202that intercepts a function call from an executing application where thefunction call specifies a target function and the target function isassociated with a target processor. The intercept module 202, in oneembodiment, is substantially similar to the intercept modules 202 of theapparatuses 200, 300 of FIGS. 2 and 3. In the embodiment, the comparisonmodule 702 compares energy consumption characteristics of the pluralityof processors in response to the intercept module 202 intercepting thefunction call.

In addition, the selection module 704 selects a function from aplurality of available functions to execute the function call where eachfunction is associated with a separate processor. The selection module704 selects the function associated with the processor with a lowestenergy consumption for execution of the function call. Also in theembodiment, the execution module 706 reroutes the function call toexecute the selected function. In the embodiment, the selection module704 may be similar to or the same as the selection module 204 and theexecution module 706 may be similar to the execution module 206described in the apparatuses 200, 300 of FIGS. 2 and 3 and the selectionmodule 704 and the execution module 706 includes functionality describedwith regard to the apparatuses 700, 800 of FIGS. 7 and 8 where selectioncriteria for a function is specified to select a most energy efficientprocessor.

FIG. 9 is a schematic flow chart diagram illustrating one embodiment ofa method 900 for determining execution of a function call based onefficiency. The method 900 begins and compares 902 energy consumptioncharacteristics of a plurality of processors available for execution ofa function, where each energy consumption characteristic varies as afunction of function size. In one embodiment, the comparison module 702compares 902 energy consumption characteristics.

The method 900 selects 904, based on the size of the function, aprocessor from the plurality of available processors 112, 116, 120, 124,128 with a lowest energy consumption for execution of the function andexecutes 906 the function on the selected processor, for example, on theGPU 116. Typically, the method 900 selects a function for the selectedprocessor (e.g. GPU 116) that is from a library for the selectedprocessor and matches a function of a function call. For example, if afunction call specifies an FFT function, the method 900 selects an FFTfunction a library for the selected processor (e.g. GPU 116). Theavailable processors 112, 116,120, 124, 128 may be limited to those thathave an associated function that matches a function specified in afunction call. For example, where a function call specifies an FFT,processors that may not have an associated FFT function may not be partof the available processors or may not be selected 904 by the method900. In some embodiments, the selection module 704 selects 904 aprocessor and the execution module 706 executes 906 the function.

FIG. 10 is a schematic flow chart diagram illustrating one embodiment ofan alternate method 1000 for determining execution of a function callbased on efficiency. The method 1000 begins and calculates 1002 energyconsumption of an executed function executed on a processor of availableprocessors 112, 116, 120, 124, 128 and correlates 1004 a size of eachexecuted function to energy consumed during execution of the function todetermine the energy consumption characteristic of the processor. Themethod 1000 updates 1006 an initial energy consumption characteristic ofa processor based on the function execution data from the previouslyexecuted functions. For example, the method 1000 updates an initialenergy consumption characteristic of a processor that includes data fromexecuted functions, from user supplied data, and/or vendor supplieddata.

The method 1000 may repeat calculating 1002 energy consumption of anexecuted function and correlating 1004 a size of each executed functionto energy consumed during execution of the function and updating 1006energy consumption characteristics each time a function executes on oneof the available processors 112, 116, 120, 124, 128. In someembodiments, the energy calculation module 804 calculates 1002 energyconsumption of an executed function and the correlation module 806correlates 1004 a size of each executed function to energy consumedduring execution of the function to determine the energy consumptioncharacteristic of the processor. In addition, the update module 808 mayupdate 1006 an initial energy consumption characteristic of a processoror data from an executed function.

The method 1000 intercepts 1008 a function call from an executingapplication where the function call specifies a target function and themethod 1000 determines 1010 a current state of the processor and adjuststhe startup cost for the processor based the current condition of theprocessor. For example, the method 1000 may determine that a processoris in a state of readiness so that an initial startup cost his higherthan a current startup cost and the method 1000 then adjusts the startupcost for the processor. Typically, the method 1000 determines 1010 thecurrent state of available processors and associated functions that arebeing considered for execution in place of a function specified in afunction call. In one embodiment, the startup module 812 determines 1010the current state of one or more processors and adjusts the startupcosts accordingly.

The method 1000 updates 1012 an initial energy consumptioncharacteristic of a processor based on the function execution data fromthe previously executed functions. The initial energy consumptioncharacteristics may be supplied by a vendor, may be supplied by a user,or may include data from previously executed functions. In one example,the update module 808 updates 1012 initial energy consumptioncharacteristics. The method 1000 compares 1014 energy consumptioncharacteristics of a plurality of processors available for execution ofa function where each energy consumption characteristic varies as afunction of function size and selects 1016, based on the size of thefunction, a processor from the plurality of processors with a lowestenergy consumption for execution of the function. The comparison module702 may compare 1014 energy consumption characteristics and theselection module 704 may select 1016 a function. The method 1000reroutes 1018 the function call to execute the selected function theselected processor, and the method 1000 ends. The execution module 706may reroute 1018 the function call. In addition, the method 1000 mayreturn after execution of the function to use data from the newlycalculated function to calculate 1002 the energy consumption of theexecuted function.

FIG. 11 is a chart depicting one embodiment of incremental energyrequirements of three processors based on workload size. The chartdepicts incremental energy on the vertical axis and workload size (i.e.function size) on the horizontal axis. The chart shows an energyconsumption characteristic of three available processors: a CPU (e.g.CPU 112), a first accelerator (e.g. accelerator 124), and a secondaccelerator (i.e. another accelerator 124). Other embodiments mayinclude more available processors. The chart in FIG. 11 depicts theenergy consumption characteristic of the CPU as a solid line that has alow startup cost (i.e. initial offset on the left side of the CPU energyconsumption characteristic). The CPU energy consumption characteristicthen increases at a relatively high rate with respect to the other twoaccelerators as workload size increases.

The chart depicts the energy consumption characteristic of the firstaccelerator as a line with long dashes separated by a single dot. Thefirst accelerator energy consumption characteristic has a relativelyhigh startup cost, characterized by a large initial offset to the leftof the curve. The first accelerator energy consumption characteristicthen increases with load size, but not as quickly as for the CPU energyconsumption characteristic. The chart depicts the second acceleratorenergy consumption characteristic as a dashed line with dashes separatedby two dots. While the startup cost for the second accelerator energyconsumption characteristic is higher than for the CPU, the slope of theline is less than for the CPU so the second accelerator energyconsumption characteristic is less than for the CPU at a certainworkload size.

The chart depicts minimum incremental energy as a function of workloadsize as a dotted line. The minimum incremental energy follows the CPUenergy consumption characteristic for smaller workloads and then followsthe second accelerator energy consumption characteristic when the secondaccelerator energy consumption characteristic is smaller than the CPUenergy consumption characteristic. The comparison module 702 may usedata similar to what is in FIG. 11 to compare the energy consumptioncharacteristics and the selection module 704 may then select a processorwith a processor with a lowest energy consumption characteristic. Forexample, the comparison module 702 may use equations, tables, etc. asenergy consumption characteristic input. Note that the update module 808may update the energy consumption characteristic of each processor basedon current conditions.

FIG. 12 is a chart depicting another embodiment of incremental energyrequirements of three processors based on workload size. The chart inFIG. 12 is similar to the chart in FIG. 11 except that the energyconsumption characteristic of the first and second accelerators differsfrom the chart in FIG. 11. In the chart of FIG. 12, the minimumincremental energy line follows the CPU energy consumptioncharacteristic for small workloads, then the second accelerator energyconsumption characteristic for midrange sized workloads and then thefirst accelerator energy consumption characteristic for largerworkloads. Again, the comparison module 702 may compare the energyconsumption characteristics as shown in FIG. 12 and the selection module704 may then select a processor with a processor with a lowest energyconsumption characteristic. Again, the update module 808 may alter theenergy consumption characteristics based on current conditions ofavailable processors.

FIG. 13 is a schematic block diagram illustrating one embodiment of anapparatus 1300 for determining execution of a function call based onperformance. The apparatus 1300 includes one embodiment, of theperformance apparatus 106 with a comparison module 1302, a selectionmodule 1304, and an execution module 1306, which are described below.While the performance apparatus 106 is depicted in the CPU memory 110 inFIG. 1 and the modules 1302, 1304, 1306 are depicted in the apparatus1300 in FIG. 13, one of skill in the art will recognize that all or aportion of the modules 1302, 1304, 1306 may be located elsewhere. Inaddition, the modules 1302, 1304, 1306 may be implemented using hardwarecircuits, a programmable logic device, executable code, or anycombination thereof. For example, the comparison module 1302 may includehardware circuits to measure energy consumption or may be implemented inexecutable code or in a programmable logic device that receives powerconsumption data. The location and makeup of the modules of theapparatus 1300 of FIG. 13 may also apply to the modules 1402, 1404,1406, 1408, 1410, 1412, 202 of the apparatus 1400 of FIG. 14.

The apparatus 1300 includes, in one embodiment, a comparison module 1302that compares performance characteristics of a plurality of processorsavailable for execution of a function where each performancecharacteristic varies as a function of function size. For example, thecomparison module 1302 may compare performance characteristics ofavailable processors, such as one of the CPUs 112, a GPU 116, an FPGA120, etc. A performance characteristic, in one embodiment, includes anestimated execution time for a function associated with an availableprocessor, where a shorter execution time may be more desirable than alonger execution time. In other embodiments, a performancecharacteristic includes other characteristics, such as a measure ofaffected resources, energy consumption, and the like. In one embodiment,a performance characteristic for a processor includes a combination ofvarious performance metrics. In one embodiment, the selection module1302 determines a performance characteristic for an available processor(e.g. GPU 116) based on a current condition of the processor. Forexample, the selection module 1302 may take into account that the GPU116 is currently executing another function. One of skill in the artwill recognize various performance metrics that may be used for aperformance characteristic of an available processor.

The apparatus 1300 includes an embodiment of a selection module 1304that selects, based on a size of the function, a processor from theplurality of processors with a best performance for execution of thefunction. For example, the comparison module 1304 may determine functionparameters, such as function size, function type, etc. from a functioncall and then compare performance from among available processors, suchas one or more of the CPUs 112, the GPU 116, the FPGA 120, etc. and theselection module 1304 may then select the GPU 116 as having the bestperformance for the function size of all of the available processors112, 116, 120, 124, 128. The selection module 1304 may be an embodimentof the selection module 204 of the apparatuses 200, 300 of FIGS. 2 and3.

The apparatus 1300 includes an embodiment of an execution module 1306that executes the function on the selected processor. For example, ifthe selection module 1304 determines that the GPU 116 is the selectedprocessor, the execution module 1306 executes an appropriate function onthe GPU 116. The function may be from a function library for the GPU 116and is a replacement for a function called out in a function call. Theexecution module 1306 may be an embodiment of the execution module 206in the apparatuses 200, 300 of FIGS. 2 and 3 where the function call isdynamically intercepted during execution of a program.

FIG. 14 is a schematic block diagram illustrating one embodiment of analternate apparatus 1400 for determining execution of a function callbased on performance. The apparatus 1400 includes another embodiment ofthe performance apparatus 106 which includes a comparison module 1302, aselection module 1304, and an execution module 1306, which aresubstantially similar to those described above in relation to theapparatus 1300 of FIG. 13. In various embodiments, the apparatus 1400may also include one or more of a performance module 1402 with aperformance calculation module 1404, a correlation module 1406, acontext module 1408, and/or an update module 1410, a format module 1412and an intercept module 202, which are described below.

The apparatus 1400, in one embodiment, includes a performance module1402 that determines the performance characteristic of each of theplurality of available processors (e.g. 112, 116, 120, 124, 128). Forexample, the performance module 1402 may evaluate performance of theavailable processors 112, 116, 120, 124, 128 based on a size of thefunction specified in a function call. The performance module 1402, inone embodiment, determines the performance characteristic of theavailable processors 112, 116, 120, 124, 128 based on vendor data, usersupplied data, data from previously executed functions, etc.

In one embodiment, the performance module 1402 includes a performancecalculation module 1404 that calculates performance of an executedfunction executed on a processor and a correlation module 1406 thatcorrelates a size of each executed function to performance duringexecution of the function to determine a performance characteristic ofthe processor. For example, the performance module 1402 may accumulatedata from executed functions and the performance calculation module 1404may then calculate performance of each executed function executed on aprocessor. The correlation module 1406 may then use results from theperformance calculation module 1404 to correlate a size of the executedfunction to the calculated performance to determine a data point in aperformance characteristic of the processor that executed the function.The performance characteristic may include amount of time to execute thefunction, execution time for one or more subroutines or parts of thefunction, energy consumption during execution of the function, etc. andmay also include a combination of performance metrics. One of skill inthe art will recognize other performance characteristics that may becalculated by the performance calculation module 1404 and thencorrelated to a function size by the correlation module 1406.

The apparatus 1400, in one embodiment, includes a context module 1408that tracks a context for each function executed on a processor. Thecomparison module 1302 then compares performance characteristics of theplurality of available processors 112, 116, 120, 124, 128 with similarcontexts. A context may include a type of data being evaluated, otherprograms running at the same time, commands in a queue, and the like.For example, an FFT function may be used for various purposes. Onecontext may be to use FFT for evaluating what objects are in aphotograph. Another context may be to use FFT to understand an image inthe frequency domain. Another context may be to use FFT for analysis ofa received wireless signal. Commands on a stack or in a queue may beused to understand a particular context.

A particular function may perform differently under different contextsand in one embodiment each context for a processor may include aseparate performance characteristic. The context module 1408 maydetermine a current context so that the comparison module 1302 comparesperformance characteristics of available processors 112, 116, 120, 128corresponding to a current context. The context module 1408 may alsodetermine a context for an executed function so that the correlationmodule 1406 correlates a current context with a calculated performancefor the executed function.

The apparatus 1400, in one embodiment, includes an update module 1410that updates an initial performance characteristic of a processor basedon the function execution data from previously executed functions. Forexample, the initial performance characteristic for a function may besupplied by a vendor or user. As functions are executed, the updatemodule 1410 may use results from the performance calculation module1404, the correlation module 1406, and the context module 1408 to modifythe initial performance characteristic. In other embodiments, theinitial performance characteristic may include both vendor or usersupplied data mixed with data from executed functions or the initialperformance characteristic may be solely from data of executed functionswhere the data from executed functions has completely overwritten datafrom a vendor or user. In some embodiments, the update module 1410 inthe apparatus 1400 of FIG. 14 may share functionality with the updatemodule 808 of the apparatus 800 of FIG. 8.

The apparatus 1400, in one embodiment, includes a function size module810 (not shown) that is substantially similar to the function sizemodule 810 of the apparatus 800 of FIG. 8. The apparatus 1400 may alsoinclude an intercept module 202 that intercepts a function call from anexecuting application where the function call specifies a targetfunction and the target function is associated with a target processor.The intercept module 202, in one embodiment, is substantially similar tothe intercept modules 202 of the apparatuses 200, 300 of FIGS. 2 and 3.In the embodiment, the comparison module 1302 compares performancecharacteristics of the plurality of processors in response to theintercept module 202 intercepting the function call.

In addition, the selection module 1304 selects a function from aplurality of available functions to execute the function call where eachfunction is associated with a separate processor. The selection module1304 selects the function associated with the processor with a bestperformance for execution of the function call. Also in the embodiment,the execution module 706 reroutes the function call to execute theselected function on the selected processor. In the embodiment, theselection module 1304 may be similar to or the same as the selectionmodules 204, 704 and the execution module 1306 may be similar to theexecution modules 206, 706 described in the apparatuses 200, 300, 700800 of FIGS. 2, 3, 7 and 8 and the selection module 1304 and theexecution module 1306 includes functionality described with regard tothe apparatuses 1300, 1400 of FIGS. 13 and 14 where selection criteriafor a function is specified to select a processor with a bestperformance.

FIG. 15 is a schematic flow chart diagram illustrating one embodiment ofa method 1500 for determining execution of a function call based onperformance. The method 1500 begins and compares 1502 performancecharacteristics of a plurality of processors available for execution ofa function (e.g. 112, 116, 120, 124, 128). Each performancecharacteristic varies as a function of function size. In one embodiment,the comparison module 1302 compares 1502 performance characteristics.The method 1500 selects 1504, based on a size of the function, aprocessor from the plurality of available processors 112, 116, 120, 124,128 with a best performance for execution of the function and the method1500 executes 1506 the function on the selected processor, and themethod 1500 ends. In some embodiments, the selection module 1304 selects1504 a processor and the execution module 1306 executes 1506 theselected function.

FIG. 16 is a schematic flow chart diagram illustrating one embodiment ofan alternate method 1600 for determining execution of a function callbased on performance. The method 1600 begins and calculates 1602performance of an executed function executed on a processor andcorrelates 1604 a size of each executed function to performance duringexecution of the function to determine a performance characteristic ofthe processor. The method 1600 correlates 1606 a context of the executedfunction to the performance of the executed function and updates 1608 aninitial performance characteristic of a processor based on the functionexecution data from previously executed functions. For example, theinitial performance characteristic for a processor may include a datapoint from a vendor or user for a particular function size that is thesame function size as the executed function.

The method 1600 updates 1608 the data point with data from the executedfunction. In various embodiments, the performance calculation module1404 may calculate 1602 performance of the executed function, thecorrelation module 1406 may correlate 1604 function size of the functionto performance during execution of the function to determine aperformance characteristic, the context module 1408 may determine 1606 acontext of the executed function, and the update module 1410 may update1608 performance characteristics of the processor that executed thefunction based on the data from the executed function.

The method 1600 intercepts 1610 a function call from an executingapplication where the function call specifies a target function and thetarget function is associated with a target processor and compares 1612performance characteristics of a plurality of processors 112, 116, 120,124, 128 available for execution of a function. The method 1600 mayinclude the target processor in the comparison 1612. The interceptmodule 202, in one embodiment, intercepts 1610 the function call and thecomparison module 1302 compares 1612 performance characteristics. Themethod 1600 selects 1614, based on size of the function of the functioncall, an available processor (e.g. the GPU 116) with a best performanceand the method 1600 reroutes 1616 and executes the function on theselected processor (e.g. 116), and the method 1600 ends. In addition,the method 1600 may return and calculate 1602 performance of the newlyexecuted function to update 1608 the performance characteristic of theprocessor (e.g. 116) that executed the function. In some embodiments,the selection module 1304 selects 1614 the processor and the executionmodule 1306 reroutes 1616 and executes the function on the selectedprocessor.

FIG. 17 is a chart depicting one embodiment of computing speed versusfast Fourier transform size of two processors. The chart showsperformance characteristics of two processors in terms of variation ofcomputing speed as a function of function size. The vertical axis of thechart is relative computing speed and the horizontal axis depicts FFTsize scaled in log base 2. The dashed line is for an accelerator (e.g.124) executing a function from the math kernel library (“MKL”) and thesolid line is for an accelerator (e.g. 124) executing a similar functionfrom the FFTW library, which is a C-language subroutine library forcomputing discrete FFT in one or more dimensions, of arbitrary inputsize, and of both real and complex data. The chart shows that bothprocessors perform equally below a relative computing speed of 1.5, butthat the processor executing a function from the FFTW library outperforms the processor executing a similar function from the MKL. Thecomparison module 1302 may use data used to plot the chart whencomparing performance data. For example, if a function has an FFT sizeof 2¹⁴, the selection module 1304 may select the processor executing theFFTW functions.

FIG. 18 is a schematic block diagram illustrating one embodiment of anapparatus 1800 for determining a memory management method. The apparatus1800 includes one embodiment of the memory apparatus 108 with a memorycondition module 1802, a memory selection module 1804 and animplementation module 1806, which are described below. While the memoryapparatus 108 is depicted in FIG. 1 in the CPU memory 110, all or aportion of the modules 1802-1806 may be located in another location. Inaddition, all or a portion of the modules 1802-1806 may be implementedin hardware. For example, the memory condition module 1802 may includehardware circuits to determine memory parameters. In other embodiments,the modules 1802-1808 are executable code, which may receive input fromand interact with hardware circuits. One of skill in the art willrecognize various ways to implement the memory apparatus 108.

The apparatus 1800, in one embodiment, includes a memory conditionmodule 1802 that determines memory parameters of host memory and devicememory prior to a device executing a function. Note that as used herein,“function” may be used interchangeably with “program,” “subroutine,” orother presently executable task. In one embodiment, the host memory ismemory accessible by a host. The host is a processor and may be a CPU,such as one of the CPUs 112 of the system 100 of FIG. 1 or may beanother processor, such as the GPU 116, the FPGA 120, etc. For example,if the host is a CPU 112, the CPU memory 110 may be the host memory.

The terms “host,” “host memory,” “device” and “device memory” are usedin conjunction with the memory apparatus 108 and associated flowchartdiagrams in FIGS. 20 and 21 as terms of art found in literaturediscussing various memory management methods. The host, in oneembodiment, is a processor that tasks the device with execution of afunction. For example, the host may be executing a program that includesa function call or may be executing a function call and the device isselected to execute a function that is compatible with a functionspecified in the function call. For example, the device may be theprocessor selected by the selection modules 204, 704, 1304 of theapparatuses 200, 300, 700, 800, 1300, 1400 of FIGS. 2, 3, 7, 8, 13, and14. Once a selection module 204, 704, 1304 has selected a processor, andthe associated execution module 206, 706, 1306 has begun or is about tobegin executing a function on the selected processor, the memorycondition module 1802 may determine memory parameters of the host memoryand the device memory.

The device memory, in one embodiment, is memory accessible to thedevice. For example, if the device is a selected processor that wasselected by a selection module 204, 704, 1304 and the selected processoris the GPU 116, the device memory may be GPU memory 118. Typically, thehost and the device each have an associated memory. However, in someembodiments, the device may rely on the host memory and may not have anydevice memory, may have device memory dedicated to other tasks, may haveinsufficient memory, etc.

In other embodiments, the host is tasked with execution of a program, asubroutine, a function, a task or some other executable code that isindependent of the apparatuses 200, 300, 700, 800, 1300, 1400 of FIGS.2, 3, 7, 8, 13, and 14, but interacts with the device to execute all ora portion of the program, subroutine, function, etc. In one embodiment,the function is specified in a function call issued from an applicationexecuting on the host.

The host memory and/or device memory have two or more available memorymanagement methods and the memory apparatus 108 is capable of selectingand changing a memory management method. For example, available memorymanagement methods may include pageable, page-locked, and one ofzero-copy, unified address space memory management methods, and thelike. Typically, data used in the execution of a program, function, etc.is stored in the host memory and/or device memory.

The pageable memory management method is common and allows the memoryapparatus 108 to store and retrieve data from secondary storage, whennecessary, such as a hard disk drive, flash memory, or othernon-volatile memory that is typically slower than the host memory or thedevice memory. Paging is often used when an amount of data used by ahost for execution of a program, function, etc. exceeds availablecapacity of the host memory and/or device memory. Often, the host memoryand/or device memory are used for multiple tasks, programs, etc. so thatnot all of the host/device memory is available to a particular program,function, etc.

A disadvantage of the pageable memory management method is that directmemory access (“DMA”) cannot be used. DMA is a method where a DMAcontroller takes over transfer of data from the host memory to thedevice memory, which frees up the host for other tasks. However, a DMAcontroller typically lacks the capability to access data paged out tothe secondary storage. Typically, DMA is efficient and is desirable, soa cost of choosing the pageable memory management method is loss of theability to use DMA. However, the pageable memory management method is agood option when required data exceeds available capacity in the hostmemory.

The page-locked memory management method is another common memorymanagement method and has data locked or pinned in the host memory sothat data is not paged out to the secondary storage. For the page-lockedmemory management method, a certain amount of host memory (typically theCPU memory 110) is allocated for use for data of an executing program,function, etc. and the data is loaded into the host memory and not pagedout so that the device (i.e. GPU 116, FPGA 120, etc.) can access thedata through a DMA process without a memory management controller of thehost being involved to retrieve paged out data back to the host memory.If the host memory was pageable, DMA typically couldn't be used becauseDMA is for accessing data in memory and the device or a DMA controllerdoes not have knowledge or capability to go get data paged out to thesecondary storage. For the page-locked memory management method,zero-copy memory management method, or unified address space memorymanagement method, the amount of available host memory space to beallocated for use by the function call typically is greater than a sizeof the data to be loaded into the host memory.

For the zero-copy memory management method, a portion of the host memoryis allocated as page-locked (e.g. pinned) memory and the allocatedmemory is mapped into the address space of the device. The device thenhas knowledge of where the data is located in the host memory, thusavoiding transferring all of the data to from the host memory to thedevice memory. For example, if zero-copy is used for an FFT function andthe FFT function needs values A and B, a memory controller for thedevice may read values A and B from the host memory instead of copyingdata to the device memory to access values A and B.

While the zero-copy memory management method is more efficient in someinstances, memory access of data in the host memory is typically slowerthan memory access by the device to the device memory. Often, PCI-E or asimilar bus is used for the data access. In a situation where aparticular host memory location is accessed multiple times, for examplewhen a calculation uses a value repeatedly, such as for a windowingfunction or convolution, zero-copy may not be as efficient as thepage-locked memory management method. Conversely, when a value isaccessed a low number of times, for instance accessed once, and in caseswhere available device memory is low or non-existent, using thezero-copy memory management method is often more efficient than usingthe page-locked or pageable memory management methods.

Performance using the unified address space memory management method issimilar to the zero-copy memory management method. One example ofimplementation of the unified address space memory management method isin CUDA 6. For unified address space, the host memory and the devicememory share a single virtual address space with a single memorypointer. Often, the unified address space performs about the same aszero-copy. An advantage of unified address space is that writingadditional software is not required for data transfer. The unifiedaddress space memory management method borrows from multi-processorcache coherency technology and the coherency protocol may be hardwarebased or implemented in software as a runtime, but is not managed by theapplication software. For example, when an application wants to use anaccelerator 124, GPU 116, etc., pageable, page-locked, and zero-copymemory management methods typically require additional lines of code.However, the unified address space memory management method typicallydoes not require extra lines of code and memory allocation is on thedevice. When memory is allocated for execution of a program, function,etc., host memory is marked as device memory in a virtual address space.The unified address space memory management method is similar in conceptto memory for a multi-processor system.

In some embodiments, use of zero-copy or unified address space is auser's choice. Unified address space, in some embodiments, worksefficiency for smaller increments of data; sizes that are related to thephysical memory lines, not related to the logical size of a working setor data structure. For the unified address space memory managementmethod, both the host and the device use the combined host memory andthe device memory as their own by way of the virtual address space.Thus, in unified address space, typically when one side writes to anaddress, there is a protocol whereby the other side is notified thatbefore that address is used an updated version is required.

The unified address space memory management method typically requirespinned data in the host memory, like zero-copy, and has similarefficiency issues as zero-copy. For example, if a value is accessedmultiple times during execution of a function, program, subroutine, etc.using unified address space may be less efficient than the page-lockedmemory management method, and if data required for execution exceedsavailable space in the host memory, the pageable memory managementmethod may be a better choice.

As stated above, the memory condition module 1802 determines memoryparameters of host memory and device memory prior to a device executinga function. The memory parameters may include amount of available hostmemory, amount of available device memory, number of times a block ofmemory is accessed, a complexity of memory allocation based on commandsin a queue, and the like. For example, the memory condition module 1802may determine an amount of available host memory to determine if datarequired for execution of a function will exceed the available hostmemory, which may be used to determine if the pageable memory managementmethod should be used or if another memory management method may be usedinstead to increase performance. The memory condition module 1802 maydetermine an amount of available device memory, which may be useful indetermining if zero-copy or unified address space may be usedefficiently during execution of a function, program, etc.

The memory condition module 1802 may determine a number of times a blockof memory will be accessed, which is useful in determining if zero-copyor unified address space may be used efficiently during execution of afunction, program, etc. The memory condition module 1802 may determinethe number of times a block of memory is accessed by looking at datafrom previously executed functions, by knowledge of a function type,etc. For example, the memory condition module 1802 may determine that afunction call is for a windowing function or a convolution, which may befunctions known to re-access data values repeatedly. In otherembodiments, the memory condition module 1802 may look at data fromexecuted functions to determine function type, context, etc. where thedata from the executed functions may include an amount of access to asame data block multiple times. Where a currently executing function issimilar to an executed function and the data from the previouslyexecuted functions indicates repeated access to a memory location, thememory condition module 1802 determine that the current function willaccess a data block multiple times.

The memory condition module 1802 may determine a complexity of memoryallocation based on commands in a queue, which may indicate that usingthe zero-copy or unified address space memory management spaces isefficient. For example, the memory condition module 1802 may determineif multiple function call routings are scheduled in a queue or happenconcurrently. In this situation, zero-copy or unified address space maybe good selections. One of skill in the art will recognize othersituations that increase complexity of memory allocation. The pageable,page-locked, zero-copy, and unified address space memory managementmethods are described herein for convenience and other memory managementmethods may also be used by the memory apparatus 108.

The apparatus 1800 includes, in one embodiment, a memory selectionmodule 1804 that selects a memory management method based on the memoryparameters. For example, the memory selection module 1804 may determinethat data for a currently executing function, program, subroutine, etc.exceeds available space in the host memory and the memory selectionmodule 1804 may select the pageable memory management method. The memoryselection module 1804, in one embodiment, changes from a current memorymanagement method to the selected memory management method. Theapparatus 1800, in one embodiment, includes an implementation module1806 that implements the selected memory management method inassociation with execution of the function, program, subroutine, etc. Inone embodiment, the memory apparatus 108 functions dynamically after theintercept module 202 intercepts a function call. In other embodiments,the memory apparatus 108 operates prior to execution of a function inconditions where a device will execute the function. One of skill in theart will recognize other suitable times for the memory apparatus 108 toevaluate various memory management methods and change from a currentmemory management method to another memory management method.

FIG. 19 is a schematic block diagram illustrating one embodiment of analternate apparatus 1900 for determining a memory management method. Theapparatus 1900 includes another embodiment of the memory apparatus 108with a memory condition module 1802, a memory selection module 1804 andan implementation module 1806, which are substantially similar to thosedescribed above in relation to the apparatus 1800 of FIG. 18. In variousembodiments, the apparatus 1900 may also include one or more of a memoryhistory module 1902 and a current condition module 1904 in the memorycondition module 1802, which are described below.

In one embodiment, the apparatus 1900 includes a memory history module1902 that determines memory parameters present during execution of oneor more previously executed functions. The memory history module 1902,in one embodiment, tracks, along with the memory parameters of anexecuted function, function type, function parameters, and/or whichdevice executed the function, where several devices were available forexecution of the function. In addition, the memory history module 1902may track other useful information, such as context of an executedfunction. The memory selection module 1804 may use the memory parametersfrom the memory history module 1902 to select a memory managementmethod.

In another embodiment, the apparatus 1900 includes a current conditionmodule 1904 that determines current memory parameters of the host memoryand the device memory. The memory selection module 1804 may use thecurrent memory parameters in selecting a memory management method. Forexample, the current condition module 1904 may determine an amount ofavailable host memory, an amount of available device memory, a size ofdata used by a function that is about to be executed, status of otherexecuting functions, programs, etc. to assess an amount of memory thatbe available once a function has been fully executed, and other currentconditions that may affect selection of a memory management method.

In one embodiment, the memory selection module 1804 selects the pageablememory management method in response to the memory condition module 1802determining that the available host memory is below a host memorythreshold. The host memory threshold, in one embodiment, is a fixedamount of memory. In another embodiment, the host memory threshold isrelated to a size of data used by the function that is about to beexecuted. In another embodiment, the host memory threshold is related tothe function. For example, each function in a library may require aparticular mount of host memory for execution or for a particular memorymanagement method. One of skill in the art will recognize other hostmemory thresholds.

In another embodiment, the memory selection module 1804 selects thepage-locked memory management method in response to the memory conditionmodule 1802 determining that the available host memory is above the hostmemory threshold and the available device memory is above a devicememory threshold, a same memory block in the host memory is accessedmore than a memory block access threshold and/or the complexity ofmemory allocation is below a complexity threshold. The device memorythreshold, in one embodiment, is related to the size of the data usedthe by the function that is about to be executed. In other embodiments,the device memory threshold is related to other parameters useful inselecting a memory management method.

The memory block access threshold may be set based on performance of afunction using the page-locked memory management method versus using thezero-copy or unified address space memory management methods. Typically,the memory block access threshold is set at a low value because repeatedaccess of a memory block typically indicates that the page-locked memorymanagement method is more efficient than the zero-copy or unifiedaddress space memory management methods.

In one embodiment, complexity of memory allocation is expressed as avalue and the complexity threshold is set at a value indicative of a lowamount of complexity of memory allocation. One of skill in the art willrecognize ways to determine complexity of memory allocation and toconvert the complexity to a value.

In another embodiment, the memory selection module 1804 selects thezero-copy memory management method or the unified address space memorymanagement method in response to the memory condition module 1802determining that the available host memory is above the host memorythreshold and the available device memory is below the device memorythreshold, a same page in the host memory is accessed less than a pageaccess threshold and/or the complexity of memory allocation is above acomplexity threshold. Where other memory management methods areavailable, other memory management method selection criteria may be usedby the memory selection module 1804 to select a memory managementmethod.

In one embodiment, each memory management method includes at least oneweighting factor and the memory condition module 1802 adjusts theweighting factor(s) of each memory management method based on thedetermined memory parameters and the memory selection module 1804selects the memory management method based on the weighting factors. Forexample, if the memory condition module 1802 determines that theavailable host memory is lower that the size of data to be used inexecuting a function, the memory condition module 1802 may increase aweighting factor for the pageable memory management method and may alsolower a weighting factor for the page-locked, zero-copy, and unitedaddress space memory management methods. Other memory parameters mayalso increase or decrease the weighting factors.

In one embodiment, each memory management method has both a historicalweighting factor and a current conditions weighting factor and thememory history module 1902 adjusts the historical weighting factor ofeach memory management method based on memory parameters present duringexecution of the previously executed functions. In the embodiment, thecurrent condition module 1904 adjusts the current conditions weightingfactor of each memory management method based on current memoryparameters of the host memory and the device memory. The memorycondition module 1802 combines the historical weighting factor and thecurrent conditions weighting factor for each memory management method.The memory selection module 1804 may then use the combined weightingfactors to select a memory management method.

FIG. 20 is a schematic flow chart diagram illustrating one embodiment ofa method 2000 for determining a memory management method. The method2000 begins and determines 2002 memory parameters of host memory anddevice memory prior to a device executing a function. In one embodiment,the memory condition module 1802 determines 2002 the memory parameters.The memory parameters may include amount of available host memory,amount of available device memory, number of times a block of memory isaccessed, a complexity of memory allocation based on commands in aqueue, and the like and may use data from previously executed functions.

The method 2000 selects 2004 a memory management method based on thememory parameters, such as the pageable, page-locked, zero-copy, orunified address space memory management methods. In one embodiment, thememory selection module 1804 selects 2004 a memory management method.The method 2000 implements 2006 the selected memory management method inassociation with execution of the function, and the method 2000 ends.For example, the method 2000 may implement 2006 the selected memorymanagement method after a function call and before the functionspecified in the function call is executed or a function selected afterintercepting a function call is executed. In one embodiment, theimplementation module 1806 implements 2006 the selected memorymanagement method.

FIG. 21A is a first part and FIG. 21B is a second part of a schematicflow chart diagram illustrating one embodiment of an alternate method2200 for determining a memory management method. The method 2100 beginsand determines 2102 memory parameters present during execution ofpreviously executed functions. For example, the memory history module1902 may determine 2102 memory parameters of the executed functions.

The method 2100, in one embodiment, intercepts 2104 a function call andselects 2106 a device for processing the function. For example, theintercept module 202 may intercept 2104 the function call prior to aselection module (e.g. 204, 704, 1304) selecting 2106 a device (e.g. anavailable processor 112, 116, 120, 124, 128) to execute a function thatis the same or similar to the function specified in the function call.For example, the method 2100 may execute in conjunction with the methods500, 600, 900, 1000, 1500, 1600 of FIGS. 5, 6, 9, 10, 15 and 16.

The method 2100 assigns 2108 a historical weighting factor to eachmemory management method based on memory parameters from the executedfunctions. The memory history module 1902 may assign 2108 the weightingfactors. The method 2100 determines 2110 current memory parameters ofthe host memory and the device memory. For example, the currentcondition module 2110 may determine 2104 current memory parameters.

The method 2100 determines 2112 (follow A on FIG. 21A to A on FIG. 21B)if the amount of available host memory is below a host memory threshold.If the method 2100 determines 2112 that the amount of available hostmemory is not below the host memory threshold, the method 2100 increases2114 a weighting factor for the pageable memory management method. Ifthe method 2100 determines 2112 that the amount of available host memoryis below the host memory threshold, the method 2100 moves on to threeother test steps (e.g. 2116, 2118, 2120). In one embodiment, the method2100 determines 2116 if the device memory is above a device memorythreshold. In another embodiment, the method 2100 determines 2118 if thesame memory block is access more than a block access threshold. Inanother embodiment, the method 2100 determines 2120 if complexity ofmemory allocation is below a complexity threshold. The method 2100 mayalso decrease a weighing factor for the page-locked, zero-copy orunified address space memory management methods.

If the method 2100 determines 2116 that the device memory is above adevice memory threshold or determines 2118 that the same memory block isaccess more than a block access threshold or determines 2120 thatcomplexity of memory allocation is below a complexity threshold, themethod 2100 increases 2122 the page-locked memory management methodweighting factor. If the method 2100 determines 2116 that the devicememory is not above a device memory threshold or determines 2118 thatthe same memory block is not accessed more than a block access thresholdor determines 2120 that complexity of memory allocation is not below acomplexity threshold, the method 2100 increases 2124 the weightingfactor of the zero-copy or unified address space memory managementmethods. The method 2100 may decrease other weighting factors when aweighting factor is increased. The method 2100 may also include one ormore formulas or algorithms to determine an amount to increase ordecrease each weighting factor based on determined conditions. Themethod 2100 may use data from previously executed functions and/orcurrent memory parameters to increase or decrease various weightingfactors.

The method 2100 selects 2126 a memory management method based on thevarious weighting factors and implements 2128 the selected memorymanagement method. For example, the memory selection module 1804 mayselect 2126 a memory management method and the implementation module1806 may implement 2128 the selected memory management method. Themethod 2100 executes 2130 the function, and the method 2100 ends. Forexample, an execution module 206, 706, 1306 may execute the function.The method 2100 may also return (follow B on FIG. 21B to B on FIG. 21A)to determine 2102 memory management parameters from the presentlyexecuted function.

Embodiments may be practiced in other specific forms. The describedembodiments are to be considered in all respects only as illustrativeand not restrictive. The scope of the invention is, therefore, indicatedby the appended claims rather than by the foregoing description. Allchanges which come within the meaning and range of equivalency of theclaims are to be embraced within their scope.

What is claimed is:
 1. An apparatus comprising: a memory conditionmodule that determines memory parameters of host memory and devicememory prior to a device executing a function; a memory selection modulethat selects a memory management method based on the memory parameters;and an implementation module that implements the selected memorymanagement method in association with execution of the function.
 2. Theapparatus of claim 1, wherein the memory implementation module changesfrom a current memory management method to the selected memorymanagement method.
 3. The apparatus of claim 1, wherein the memorymanagement methods comprise pageable, page-locked, and one of zero-copyand unified address space memory management methods.
 4. The apparatus ofclaim 3, wherein the memory parameters comprise one or more of amount ofavailable host memory, amount of available device memory, number oftimes a block of memory is accessed, and a complexity of memoryallocation based on commands in a queue.
 5. The apparatus of claim 4,wherein the memory selection module selects: the pageable memorymanagement method in response to the memory condition module determiningthat the available host memory is below a host memory threshold; thepage-locked memory management method in response to the memory conditionmodule determining that the available host memory is above the hostmemory threshold and one or more of the available device memory is abovea device memory threshold, a same memory block in the host memory isaccessed more than a memory block access threshold and the complexity ofmemory allocation is below a complexity threshold; and one of thezero-copy memory management method and the unified address space memorymanagement method in response to the memory condition module determiningthat the available host memory is above the host memory threshold andone or more of the available device memory is below the device memorythreshold, a same memory block in the host memory is accessed less thana memory block access threshold and the complexity of memory allocationis above a complexity threshold.
 6. The apparatus of claim 1, whereinthe memory condition module further comprises one or more of: a memoryhistory module that determines memory parameters present duringexecution of one or more executed functions; and a current conditionmodule that determines current memory parameters of the host memory andthe device memory.
 7. The apparatus of claim 6, wherein the memoryhistory module tracks, along with the memory parameters of an executedfunction, one or more of function type, function parameters, and whichdevice executed the function, wherein a plurality of devices wereavailable for execution of the function.
 8. The apparatus of claim 6,wherein each memory management method comprises a weighting factor andthe memory condition module further adjusts the weighting factor of eachmemory management method based on the determined memory parameters andthe memory selection module selects the memory management method basedon the weighting factors.
 9. The apparatus of claim 8, wherein eachmemory management method comprises a historical weighting factor and acurrent conditions weighting factor and the memory history moduleadjusts the historical weighting factor of each memory management methodbased on memory parameters present during execution of the one or morepreviously executed functions and the current condition module adjuststhe current conditions weighting factor of each memory management methodbased on current memory parameters of the host memory and the devicememory, wherein the memory condition module combines the historicalweighting factor and the current conditions weighting factor for eachmemory management method.
 10. The apparatus of claim 1, wherein thefunction is specified in a function call issued from an applicationexecuting on the host.
 11. The apparatus of claim 1, wherein the devicecomprises one of a plurality of devices available for execution of thefunction.
 12. The apparatus of claim 11, wherein the memory conditionmodule determines the memory parameters in response to selection of adevice for execution of the function.
 13. A method comprising:determining memory parameters of host memory and device memory prior toa device executing a function; selecting a memory management methodbased on the memory parameters; and implementing the selected memorymanagement method in association with execution of the function.
 14. Themethod of claim 13, wherein the memory management methods comprisepageable, page-locked, and one of zero-copy and unified address spacememory management methods, and wherein the memory parameters compriseone or more of amount of available host memory, amount of availabledevice memory, number of times a page within the host memory isaccessed, and a complexity of memory allocation based on commands in aqueue.
 15. The method of claim 14, wherein selecting a memory managementmethod comprises selecting: the pageable memory management method inresponse to the memory condition module determining that the availablehost memory is below a host memory threshold; the page-locked memorymanagement method in response to the memory condition module determiningthat the available host memory is above the host memory threshold andone or more of the available device memory is above a device memorythreshold, a same page in the host memory is accessed more than a pageaccess threshold and the complexity of memory allocation is below acomplexity threshold; and one of the zero-copy memory management methodand the unified address space memory management method in response tothe memory condition module determining that the available host memoryis above the host memory threshold and one or more of the availabledevice memory is below the device memory threshold, a same page in thehost memory is accessed less than a page access threshold and thecomplexity of memory allocation is above a complexity threshold.
 16. Themethod of claim 13, further comprising: determining memory parameterspresent during execution of a plurality of previously executedfunctions; and determining current memory parameters of the host memoryand the device memory.
 17. The method of claim 16, wherein determiningmemory parameters present during execution of a plurality of previouslyexecuted functions comprises tracking, along with the memory parametersof an executed function, one or more of function type, functionparameters, and which device executed the function, wherein a pluralityof devices were available for execution of the function.
 18. The methodof claim 16, wherein each memory management method comprises a weightingfactor and determining memory parameters further comprises adjusting theweighting factor of each memory management method based on thedetermined memory parameters and selecting a memory management methodcomprises selecting the memory management method based on the weightingfactors.
 19. The method of claim 18, wherein each memory managementmethod comprises a historical weighting factor and a current conditionsweighting factor and determining memory parameters present duringexecution of a plurality of previously executed functions comprisesadjusting the historical weighting factor of each memory managementmethod based on memory parameters present during execution of theplurality of previously executed functions and determining currentmemory parameters of the host memory and the device memory comprisesadjusting the current conditions weighting factor of each memorymanagement method based on current memory parameters of the host memoryand the device memory, wherein determining the memory parameterscomprises combining the historical weighting factor and the currentconditions weighting factor for each memory management method.
 20. Asystem comprising: a host comprising a central processing unit; a hostmemory accessible by the host; a device comprising a processor; aprocessor memory accessible by the device; a communication bus betweenone or more of the host, the host memory, the device and the devicememory; a memory condition module that determines memory parameters ofthe host memory and the device memory prior to the device executing afunction; a memory selection module that selects a memory managementmethod based on the memory parameters; and a memory implementationmodule that implements the selected memory management method inassociation with execution of the function.